Gaze-Based Shared Autonomy Framework With Real-Time Action Primitive Recognition for Robot Manipulators.

Robots capable of robust, real-time recognition of human intent during manipulation tasks could be used to enhance human-robot collaboration for activities of daily living. Eye gaze-based control interfaces offer a non-invasive way to infer intent and reduce the cognitive burden on operators of complex robots. Eye gaze is traditionally used for "gaze triggering" (GT) in which staring at an object, or sequence of objects, triggers pre-programmed robotic movements. We propose an alternative approach: a neural network-based "action prediction" (AP) mode that extracts gaze-related features to recognize, and often predict, an operator's intended action primitives. We integrated the AP mode into a shared autonomy framework capable of 3D gaze reconstruction, real-time intent inference, object localization, obstacle avoidance, and dynamic trajectory planning. Using this framework, we conducted a user study to directly compare the performance of the GT and AP modes using traditional subjective performance metrics, such as Likert scales, as well as novel objective performance metrics, such as the delay of recognition. Statistical analyses suggested that the AP mode resulted in more seamless robotic movement than the state-of-the-art GT mode, and that participants generally preferred the AP mode.

Elastohydrodynamic friction of robotic and human fingers on soft micropatterned substrates.

Frictional sliding between patterned surfaces is of fundamental and practical importance in the haptic engineering of soft materials. In emerging applications such as remote surgery and soft robotics, thin fluid films between solid surfaces lead to a multiphysics coupling between solid deformation and fluid dissipation. Here, we report a scaling law that governs the peak friction values of elastohydrodynamic lubrication on patterned surfaces. These peaks, absent in smooth tribopairs, arise due to a separation of length scales in the lubricant flow. The framework is generated by varying the geometry, elasticity and fluid properties of soft tribopairs and measuring the lubricated friction with a triborheometer. The model correctly predicts the elastohydrodynamic lubrication friction of a bioinspired robotic fingertip and human fingers. Its broad applicability can inform the future design of robotic hands or grippers in realistic conditions, and open up new ways of encoding friction into haptic signals.

Toward Shared Autonomy Control Schemes for Human-Robot Systems: Action Primitive Recognition Using Eye Gaze Features.

The functional independence of individuals with upper limb impairment could be enhanced by teleoperated robots that can assist with activities of daily living. However, robot control is not always intuitive for the operator. In this work, eye gaze was leveraged as a natural way to infer human intent and advance action recognition for shared autonomy control schemes. We introduced a classifier structure for recognizing low-level action primitives that incorporates novel three-dimensional gaze-related features. We defined an action primitive as a triplet comprised of a verb, target object, and hand object. A recurrent neural network was trained to recognize a verb and target object, and was tested on three different activities. For a representative activity (making a powdered drink), the average recognition accuracy was 77% for the verb and 83% for the target object. Using a non-specific approach to classifying and indexing objects in the workspace, we observed a modest level of generalizability of the action primitive classifier across activities, including those for which the classifier was not trained. The novel input features of gaze object angle and its rate of change were especially useful for accurately recognizing action primitives and reducing the observational latency of the classifier.

Perception of Tactile Directionality via Artificial Fingerpad Deformation and Convolutional Neural Networks.

Humans can perceive tactile directionality with angular perception thresholds of 14-40° via fingerpad skin displacement. Using deformable, artificial tactile sensors, the ability to perceive tactile directionality was developed for a robotic system to aid in object manipulation tasks. Two convolutional neural networks (CNNs) were trained on tactile images created from fingerpad deformation measurements during perturbations to a handheld object. A primary CNN regression model provided a point estimate of tactile directionality over a range of grip forces, perturbation angles, and perturbation speeds. A secondary CNN model provided a variance estimate that was used to determine uncertainty about the point estimate. A 5-fold cross-validation was performed to evaluate model performance. The primary CNN produced tactile directionality point estimates with an error rate of 4.3% for a 20° angular resolution and was benchmarked against an open-source force estimation network. The model was implemented in real-time for interactions with an external agent and the environment with different object shapes and widths. The perception of tactile directionality could be used to enhance the situational awareness of human operators of telerobotic systems and to develop decision-making algorithms for context-appropriate responses by semi-autonomous robots.

Exploiting Three-Dimensional Gaze Tracking for Action Recognition During Bimanual Manipulation to Enhance Human-Robot Collaboration.

Human-robot collaboration could be advanced by facilitating the intuitive, gaze-based control of robots, and enabling robots to recognize human actions, infer human intent, and plan actions that support human goals. Traditionally, gaze tracking approaches to action recognition have relied upon computer vision-based analyses of two-dimensional egocentric camera videos. The objective of this study was to identify useful features that can be extracted from three-dimensional (3D) gaze behavior and used as inputs to machine learning algorithms for human action recognition. We investigated human gaze behavior and gaze-object interactions in 3D during the performance of a bimanual, instrumental activity of daily living: the preparation of a powdered drink. A marker-based motion capture system and binocular eye tracker were used to reconstruct 3D gaze vectors and their intersection with 3D point clouds of objects being manipulated. Statistical analyses of gaze fixation duration and saccade size suggested that some actions (pouring and stirring) may require more visual attention than other actions (reach, pick up, set down, and move). 3D gaze saliency maps, generated with high spatial resolution for six subtasks, appeared to encode action-relevant information. The "gaze object sequence" was used to capture information about the identity of objects in concert with the temporal sequence in which the objects were visually regarded. Dynamic time warping barycentric averaging was used to create a population-based set of characteristic gaze object sequences that accounted for intra- and inter-subject variability. The gaze object sequence was used to demonstrate the feasibility of a simple action recognition algorithm that utilized a dynamic time warping Euclidean distance metric. Averaged over the six subtasks, the action recognition algorithm yielded an accuracy of 96.4%, precision of 89.5%, and recall of 89.2%. This level of performance suggests that the gaze object sequence is a promising feature for action recognition whose impact could be enhanced through the use of sophisticated machine learning classifiers and algorithmic improvements for real-time implementation. Robots capable of robust, real-time recognition of human actions during manipulation tasks could be used to improve quality of life in the home and quality of work in industrial environments.

Functional Contour-following via Haptic Perception and Reinforcement Learning.

Many tasks involve the fine manipulation of objects despite limited visual feedback. In such scenarios, tactile and proprioceptive feedback can be leveraged for task completion. We present an approach for real-time haptic perception and decision-making for a haptics-driven, functional contour-following task: the closure of a ziplock bag. This task is challenging for robots because the bag is deformable, transparent, and visually occluded by artificial fingertip sensors that are also compliant. A deep neural net classifier was trained to estimate the state of a zipper within a robot's pinch grasp. A Contextual Multi-Armed Bandit (C-MAB) reinforcement learning algorithm was implemented to maximize cumulative rewards by balancing exploration versus exploitation of the state-action space. The C-MAB learner outperformed a benchmark Q-learner by more efficiently exploring the state-action space while learning a hard-to-code task. The learned C-MAB policy was tested with novel ziplock bag scenarios and contours (wire, rope). Importantly, this work contributes to the development of reinforcement learning approaches that account for limited resources such as hardware life and researcher time. As robots are used to perform complex, physically interactive tasks in unstructured or unmodeled environments, it becomes important to develop methods that enable efficient and effective learning with physical testbeds.

Medical robotics-Regulatory, ethical, and legal considerations for increasing levels of autonomy.

The regulatory, ethical, and legal barriers imposed on medical robots necessitate careful consideration of different levels of autonomy, as well as the context for use.

Discriminability of Single and Multichannel Intracortical Microstimulation within Somatosensory Cortex.

The addition of tactile and proprioceptive feedback to neuroprosthetic limbs is expected to significantly improve the control of these devices. Intracortical microstimulation (ICMS) of somatosensory cortex is a promising method of delivering this sensory feedback. To date, the main focus of somatosensory ICMS studies has been to deliver discriminable signals, corresponding to varying intensity, to a single location in cortex. However, multiple independent and simultaneous streams of sensory information will need to be encoded by ICMS to provide functionally relevant feedback for a neuroprosthetic limb (e.g., encoding contact events and pressure on multiple digits). In this study, we evaluated the ability of an awake, behaving non-human primate (Macaca mulatta) to discriminate ICMS stimuli delivered on multiple electrodes spaced within somatosensory cortex. We delivered serial stimulation on single electrodes to evaluate the discriminability of sensations corresponding to ICMS of distinct cortical locations. Additionally, we delivered trains of multichannel stimulation, derived from a tactile sensor, synchronously across multiple electrodes. Our results indicate that discrimination of multiple ICMS stimuli is a challenging task, but that discriminable sensory percepts can be elicited by both single and multichannel ICMS on electrodes spaced within somatosensory cortex.

Surgical Hand Tracking in Open Surgery Using a Versatile Motion Sensing System: Are We There Yet?

With changes in work hour limitations, there is an increasing need for objective determination of technical proficiency. Electromagnetic hand-motion analysis has previously shown only time to completion and number of movements to correlation with expertise. The present study was undertaken to evaluate the efficacy of hand-motion-tracking analysis in determining surgical skill proficiency. A nine-degree-of-freedom sensor was used and mounted on the superior aspect of a needle driver. A one-way analysis of variance and Welch's t test were performed to evaluate significance between subjects. Four Novices, four Trainees, and three Experts performed a large vessel patch anastomosis on a phantom tissue. Path length, total number of movements, absolute velocity, and total time were analyzed between groups. Compared to the Novices, Expert subjects exhibited significantly decreased total number of movements, decreased instrument path length, and decreased total time to complete tasks. There were no significant differences found in absolute velocity between groups. In this pilot study, we have identified significant differences in patterns of motion between Novice and Expert subjects. These data warrant further analysis for its predictive value in larger cohorts at different levels of training and may be a useful tool in competence-based training paradigms in the future.

Independent digit contributions to rotational manipulation in a three-digit pouring task requiring dynamic stability.

Many activities of daily living involve multi-digit, voluntary rotational manipulations of grasped objects. Yet, only a few studies have focused on coordination of individual fingertip forces during such tasks. The objective of this study was to investigate individual digit contributions to a three-digit task in which an object was rotationally manipulated against gravity. Center of mass was varied through the use of containers shaped like a water bottle, pint glass, and cocktail glass, from which subjects poured fluid carefully into a nearby receptacle. The center of mass of the grasped object changed continuously as fluid was poured out. Self-selected digit placement and contributions of fingertip forces to rotational manipulation were dependent upon anticipated center of mass location associated with container shape. The thumb resisted the rotation of the top-heavy, cocktail glass container until 79 % of the pouring phase had elapsed, but actively assisted the rotation of the less challenging containers. More directly opposing the thumb, the index finger contributed more to grasp stability. The middle finger contributed more to rotation of the container for pouring. It was found that the thumb, index, and middle fingers acted in unison temporally, but contributed independently to the grip forces and stabilizing moments throughout the dynamic, rotational manipulation task.

A robot hand testbed designed for enhancing embodiment and functional neurorehabilitation of body schema in subjects with upper limb impairment or loss.

Many upper limb amputees experience an incessant, post-amputation "phantom limb pain" and report that their missing limbs feel paralyzed in an uncomfortable posture. One hypothesis is that efferent commands no longer generate expected afferent signals, such as proprioceptive feedback from changes in limb configuration, and that the mismatch of motor commands and visual feedback is interpreted as pain. Non-invasive therapeutic techniques for treating phantom limb pain, such as mirror visual feedback (MVF), rely on visualizations of postural changes. Advances in neural interfaces for artificial sensory feedback now make it possible to combine MVF with a high-tech "rubber hand" illusion, in which subjects develop a sense of embodiment with a fake hand when subjected to congruent visual and somatosensory feedback. We discuss clinical benefits that could arise from the confluence of known concepts such as MVF and the rubber hand illusion, and new technologies such as neural interfaces for sensory feedback and highly sensorized robot hand testbeds, such as the "BairClaw" presented here. Our multi-articulating, anthropomorphic robot testbed can be used to study proprioceptive and tactile sensory stimuli during physical finger-object interactions. Conceived for artificial grasp, manipulation, and haptic exploration, the BairClaw could also be used for future studies on the neurorehabilitation of somatosensory disorders due to upper limb impairment or loss. A remote actuation system enables the modular control of tendon-driven hands. The artificial proprioception system enables direct measurement of joint angles and tendon tensions while temperature, vibration, and skin deformation are provided by a multimodal tactile sensor. The provision of multimodal sensory feedback that is spatiotemporally consistent with commanded actions could lead to benefits such as reduced phantom limb pain, and increased prosthesis use due to improved functionality and reduced cognitive burden.

Spatial asymmetry in tactile sensor skin deformation aids perception of edge orientation during haptic exploration.

Upper-limb amputees rely primarily on visual feedback when using their prostheses to interact with others or objects in their environment. A constant reliance upon visual feedback can be mentally exhausting and does not suffice for many activities when line-of-sight is unavailable. Upper-limb amputees could greatly benefit from the ability to perceive edges, one of the most salient features of 3D shape, through touch alone. We present an approach for estimating edge orientation with respect to an artificial fingertip through haptic exploration using a multimodal tactile sensor on a robot hand. Key parameters from the tactile signals for each of four exploratory procedures were used as inputs to a support vector regression model. Edge orientation angles ranging from -90 to 90 degrees were estimated with an 85-input model having an R (2) of 0.99 and RMS error of 5.08 degrees. Electrode impedance signals provided the most useful inputs by encoding spatially asymmetric skin deformation across the entire fingertip. Interestingly, sensor regions that were not in direct contact with the stimulus provided particularly useful information. Methods described here could pave the way for semi-autonomous capabilities in prosthetic or robotic hands during haptic exploration, especially when visual feedback is unavailable.

Precision grip responses to unexpected rotational perturbations scale with axis of rotation.

It has been established that rapid, pulse-like increases in precision grip forces ("catch-up responses") are elicited by unexpected translational perturbations and that response latency and strength scale according to the direction of linear slip relative to the hand as well as gravity. To determine if catch-up responses are elicited by unexpected rotational perturbations and are strength-, axis-, and/or direction-dependent, we imposed step torque loads about each of two axes which were defined relative to the subject's hand: the distal-proximal axis away from and towards the subject's palm, and the grip axis which connects the two fingertips. Precision grip responses were dominated initially by passive mechanics and then by active, unimodal catch-up responses. First dorsal interosseous activity, marking the start of the catch-up response, began 71-89 ms after the onset of perturbation. The onset latency, shape, and duration (217-231 ms) of the catch-up response were not affected by the axis, direction, or magnitude of the rotational perturbation, while strength was scaled by axis of rotation and slip conditions. Rotations about the grip axis that tilted the object away from the palm and induced rotational slip elicited stronger catch-up responses than rotations about the distal-proximal axis that twisted the object between the digits. To our knowledge, this study is the first to investigate grip responses to unexpected torque loads and to show characteristic, yet axis-dependent, catch-up responses for conditions other than pure linear slip.

Maximal voluntary fingertip force production is not limited by movement speed in combined motion and force tasks.

Numerous studies of limbs and fingers propose that force-velocity properties of muscle limit maximal voluntary force production during anisometric tasks, i.e., when muscles are shortening or lengthening. Although this proposition appears logical, our study on the simultaneous production of fingertip motion and force disagrees with this commonly held notion. We asked eight consenting adults to use their dominant index fingertip to maximize voluntary downward force against a horizontal surface at specific postures (static trials), and also during an anisometric "scratching" task of rhythmically moving the fingertip along a 5.8 +/- 0.5 cm target line. The metronome-timed flexion-extension movement speed varied 36-fold from "slow" (1.0 +/- 0.5 cm/s) to "fast" (35.9 +/- 7.8 cm/s). As expected, maximal downward voluntary force diminished (44.8 +/- 15.6%; p = 0.001) when any motion (slow or fast) was added to the task. Surprisingly, however, a 36-fold increase in speed did not affect this reduction in force magnitude. These remarkable results for such an ordinary task challenge the dominant role often attributed to force-velocity properties of muscle and provide insight into neuromechanical interactions. We propose an explanation that the simultaneous enforcement of mechanical constraints for motion and force reduces the set of feasible motor commands sufficiently so that force-velocity properties cease to be the force-limiting factor. While additional work is necessary to reveal the governing mechanisms, the dramatic influence that the simultaneous enforcement of motion and force constraints has on force output begins to explain the vulnerability of dexterous function to development, aging, and even mild neuromuscular pathology.

Improving the fitness of high-dimensional biomechanical models via data-driven stochastic exploration.

The field of complex biomechanical modeling has begun to rely on Monte Carlo techniques to investigate the effects of parameter variability and measurement uncertainty on model outputs, search for optimal parameter combinations, and define model limitations. However, advanced stochastic methods to perform data-driven explorations, such as Markov chain Monte Carlo (MCMC), become necessary as the number of model parameters increases. Here, we demonstrate the feasibility and, what to our knowledge is, the first use of an MCMC approach to improve the fitness of realistically large biomechanical models. We used a Metropolis-Hastings algorithm to search increasingly complex parameter landscapes (3, 8, 24, and 36 dimensions) to uncover underlying distributions of anatomical parameters of a "truth model" of the human thumb on the basis of simulated kinematic data (thumbnail location, orientation, and linear and angular velocities) polluted by zero-mean, uncorrelated multivariate Gaussian "measurement noise." Driven by these data, ten Markov chains searched each model parameter space for the subspace that best fit the data (posterior distribution). As expected, the convergence time increased, more local minima were found, and marginal distributions broadened as the parameter space complexity increased. In the 36-D scenario, some chains found local minima but the majority of chains converged to the true posterior distribution (confirmed using a cross-validation dataset), thus demonstrating the feasibility and utility of these methods for realistically large biomechanical problems.

Reported anatomical variability naturally leads to multimodal distributions of Denavit-Hartenberg parameters for the human thumb.

A realistic biomechanical thumb model would elucidate the functional consequences of orthopedic and neurological diseases and their treatments. We investigated whether a single parametric kinematic model can represent all thumbs, or whether different kinematic model structures are needed to represent different thumbs. We used Monte Carlo simulations to convert the anatomical variability in the kinematic model parameters into distributions of Denavit-Hartenberg parameters amenable for robotics-based models. Upon convergence (3550 simulations, where mean and coefficient of variance changed < 1% for the last 20 + % simulations) the distributions of Denavit-Hartenberg parameters appeared multimodal, in contrast to the reported unimodal distributions of the anatomy-based parameters. Cluster analysis and one-way analysis of variance confirmed four types of kinematic models (p < 0.0001) differentiated primarily by the biomechanically relevant order of MCP joint axes (in 65.2% of models, the flexion-extension axis was distal to the adduction-abduction axis); and secondarily by a detail specifying the direction of a common normal between successive axes of rotation. Importantly, this stochastic analysis of anatomical variability redefines the debate on whether a single generic biomechanical model can represent the entire population, or if subject-specific models are necessary. We suggest a practical third alternative: that anatomical and functional variability can be captured by a finite set of model-types.